A storage controller can be used as a storage server to store, manage, and retrieve data from storage devices on behalf of one or more clients on a network. The storage devices can include, for example, magnetic or optical storage-based disks or tapes, or solid state devices. Some storage controllers are designed to service file-level requests from clients, as is commonly the case with file servers used in a network attached storage (NAS) environment. Other storage controllers are designed to service block-level requests from clients, as with storage controllers used in a storage area network (SAN) environment. Still other storage controllers are capable of servicing both file-level requests and block-level requests, as is the case with certain storage controllers made by NetApp, Inc. of Sunnyvale, Calif.
A group of storage devices (e.g., disks) and one or more storage controllers assembled in a rack, or other similar enclosure, are conventionally interconnected for communication via a communication fabric. Such an assemblage can be referred to as a storage system. One purpose of the communication fabric in the storage system is to provide fast and reliable communication between the storage controller and storage devices. For modularity, the storage devices are often built into separate enclosures in the rack called “shelves.” Each shelf essentially contains only storage devices and little or no electronic “intelligence.”
Conventional communication fabrics used within racks are implemented with, for example, Fibre Channel technology. Fibre Channel communication fabric topologies describe how a number of ports of the storage controllers and storage devices are connected together. A “port” in Fibre Channel terminology is any entity that actively communicates over the Fibre Channel communication fabric. Topologies include point-to-point, in which only two devices are connected; arbitrated loop (e.g., Fibre Channel Arbitrated Loop, or “FCAL”), in which all devices are in a logical loop or ring, utilizing Fibre Channel Arbitrated Loop switches; and switched fabric, in which all devices or loops of devices are connected to Fibre Channel Fabric switches. Fibre Channel Arbitrated Loop switches have limited scalability. Fibre Channel Fabric switches are designed for long distance data center communication, and are therefore typically expensive. As such, Fibre Channel technology has certain drawbacks.
At least partially in response to the costs and complexities of Fibre Channel technologies and related storage technologies, various simpler and lower-cost technologies have more recently begun to be used. For example, direct-attach storage devices, including Serial-Attached ATA (“SATA”) and Serial-Attached SCSI (“Small Computer System Interface”) devices, or “SAS” devices, have begun to be used in storage systems. Direct-attach storage devices are designed to be directly attached to storage controllers without the use of intervening network fabrics or fabric protocols. In the case of SAS storage devices, expanders are typically interposed between SAS storage devices and a storage controller to allow for expanded connectivity (thus the term “expander”). Typically, the interconnect topologies used with expanders between SAS storage devices and storage controllers consist of either a single expander (one storage controller port and multiple disk drives directly attached to the expander), or multiple expanders cascaded to form a simple string of disk drives. In either case, the conventional use of expanders is limited to providing very basic interconnect topologies.